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The primary concerns of this paper are twofold: to understand the economic value of storage in the presence 
of ramp constraints and exogenous electricity prices, and to understand the implications of the associated 
optimal storage management policy on qualitative and quantitative characteristics of storage response to 
real-time prices. We present an analytic characterization of the optimal policy, along with the associated 
finite-horizon time-averaged value of storage. We also derive an analytical upperbound on the infinite-horizon 
time-averaged value of storage. This bound is valid for any achievable realization of prices when the support 
of the distribution is fixed, and highlights the dependence of the value of storage on ramp constraints and 
storage capacity. While the value of storage is a non-decreasing function of price volatility, due to the finite 
ramp rate, the value of storage saturates quickly as the capacity increases, regardless of volatility. To study 
the implications of the optimal policy, we first present computational experiments that suggest that optimal 
utilization of storage can, in expectation, induce a considerable amount of price elasticity near the average 
price, but little or no elasticity far from it. We then present a computational framework for understanding 
the behavior of storage as a function of price and the amount of stored energy, and for characterization of 
the buy/sell phase transition region in the price-state plane. Finally, we study the impact of market-based 
operation of storage on the required reserves, and show that the reserves may need to be expanded to 
accommodate market-based storage. 
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1. Introduction 

The growing demand for electricity and the urge to reduce greenhouse emissions promote 
large-scale integration of renewable energy sources, as well as new storage and demand- 
response technologies to improve energy efficiency in the future grid. However, renewable 
energy sources are highly uncertain and intermittent. While energy storage technologies 
can help mitigate the intermittency and narrow the gap between generation from renewable 
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resources and consumption, they may also add to the uncertainty in the system since 
the optimal response of storage to prices is a complicated function of both price and the 
amount of stored energy Modeling and understanding the behavior of storage in response 
to real-time market prices is therefore critical for reliable operation of power systems with 
large amounts of storage. 

Energy storage has a clear environmental value; it helps mitigate the intermittency of 
the renewable resources and thereby maximize their utilization. With sufficient storage 
capacity, there is no need to curtail generation from renewable energy sources when there 
is excess of it. Moreover, with proper control policies, storage can help incorporate more 
renewable resources without compromising reliability. Access to storage also reduces the 
risk associated with making advance commitments for a renewable generation owner. For 
instance, if the energy generated by the renewable source falls short of the committed 
level, the operator can compensate for the shortfall by extracting from the storage. Despite 
all these potential advantages of storage, if the economic value of storage as an arbitrage 
mechanism is not attractive, the markets may not invest sufficiently in storage. Hence, 
unless proper incentives and pricing policies are in place, the environmental and reliability 
values of storage might not materialize due to underinvestment. Therefore, there is a need 
for development of econometric models and characterization of the associated optimal 
policies that can be used for assessing the economic value of storage. This paper seeks 
to provide such characterization by presenting a model for optimal utilization of ramp- 
constrained storage in response to stochastically varying electricity prices. 

Availability of econometric models of storage and characterization of the effects of storage 
on the overall price elasticity of demand is also important for system operators who need 



to maintain stability, and guarantee reliability of the system. It was shown in Roozbehani 



Dahleh, and Mitter (2011) that in power grids with information asymmetry between 



consumers, producers, and system operators, robustness of the system to disturbances 
is greatly affected by consumers' real-time valuation of electricity, and their response to 
real-time prices. It was shown that under real-time pricing, high price volatility can be 
associated with uncertainty in demand response and high price elasticity of demand (PED). 

The existing literature covering various dimensions of storage management is extensive. 
The two major streams of literature on storage that are closely related to this work are 
the commodity trading/ warehousing literature and the electricity storage literature. The 
warehouse problem is a classical problem in the trading and commercial management of 



commodities, and has been studied in the literature extensively. As early as 1948, Cahn 



(1948) introduced the problem of optimizing purchase, i.e. injecting into storage, and sale, 
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i.e., withdrawing from storage, for the case of a warehouse with fixed size and an ini- 



tial stock of a certain commodity. Bellman (1956) formulated the warehouse problem in 
a dynamic programming framework, and Dreyfus (1957) showed that for deterministic 
prices, the optimal policy at each stage is to either fill up the storage, or empty it, or 



do nothing, depending on the stage price. Charnes, Dreze, and Miller (1966) solved the 
warehouse problem for the case of stochastic prices and showed that the optimal policy 
is the same in the stochastic case (either fill up the storage, or empty it, or do nothing). 
However, in the warehouse problem studied by these works, there is no limit on the amount 
that can be injected into, or withdrawn from storage at each stage. 



Rempala (1994), and more recently, Secomandi (2010), extended these results for the 



discrete-time case by imposing a limit on the amount that can be injected into or with- 



drawn from storage. Kaminski, Feng, and Pang (2008) solved the same extension of the 
warehouse problem in continuous time. Some other works have also considered this exten- 



sion of the warehouse problem, including Wu, Wang, and Qin (2011), who propose a 
heuristic for optimization of seasonal energy storage operations in the presence of ramp 



constraints, as well as Devalkar, Anupindi, and Sinha (2011) who seek to optimize pro- 
curement, processing, and trading of commodities in a multi-period setting. Although the 
above-mentioned references and our work share similarities in the structure of the optimal 
policy and the associated value function, the differences in the assumptions on the stochas- 
tic price process make the analytical results of these papers different from one another. 
Unlike all the previous works in this area, we assume that the price at each time period is 
independent of previous prices. Under this assumption we derive explicit formulas (recur- 
sive and/or closed- form) for the thresholds of the optimal policy and for the economic value 
of storage. We justify our assumption on prices by testing the performance of our optimal 



policy against price data from real-time markets of the PJM Interconnection (2011) and 



ISO New England (2011). Another justification for this assumption is that in practice, 



empirical estimation of conditional distributions (needed for a Markovian price model) 
requires significant amounts of data. Back of the envelop calculations show that collecting 
this much data would require going too far back in the price history. However, due to 
non-stationarity, doing so would make the data irrelevant. Although one can resort to cal- 
ibrated models for estimation of correlation in data, or try to learn the thresholds directly, 
we do not pursue these directions in this paper. In addition, we derive an upperbound 
on the optimal average value per stage of storage in the infinite-horizon case, explicitly 
highlighting its dependence on storage capacity and the ramp rate. Also different from 
previous works, we present a computational framework for estimating the average PED 
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obtained from averaging out the stage-dependence and internal state of storage. We extend 
our study of the PED induced by storage by addressing the PED as a function of the 
internal state of storage. This is particularly important in the context of electricity markets 
because the aggregate price elasticity of demand can affect price volatility and sensitiv- 



ity to disturbances (Roozbehani, Dahleh, and Mitter 2011). We also highlight another 
important aspect of state dependence of storage response by showing that under certain 
assumptions, if the system operator does not know the exact state of storage, the reserves 
may need to be expanded to accommodate market-based operation of storage. 



The literature focusing particularly on electricity storage is also extensive. Bannister 



and Kaye (1991) studied optimal management of a single storage connected to a general 



linear memoryless system in the presence of ramp constraints. However, in their model, 



the objective function is deterministic and the cost is known a priori. Also, Lee and Chen 



(1995) considered industrial consumers with time-of-use rates and used dynamic program- 
ming to determine optimal contracts and optimal sizes of battery storage systems for such 
consumers. Their work, like ours, pays special attention to the economic value of stor- 
age; however, they use a deterministic approach and relax ramp constraints. Several other 
works have studied the impacts of energy storage on the economics of integration of renew- 
able sources. A renewable generation owner can connect a storage device to the renewable 
source to optimize the overall profit over time by deciding how much energy to commit to 



sell at each stage in the time horizon. Some more recent works such as Brown, Lopes, and 



Matos (2008), Gonzalez et al. (2008), and Korpaas, Holen, and Hildrum (2003) approach 



this problem by deterministically solving this optimization problem for particular finite 
sample paths and then averaging the results of these paths. Their approach, as mentioned 
in Harsha and Dahleh (2011 ) and Kim and Powell (2011 ), does not give an optimal policy 



that can be used in practice because their policy depends on the sample path. 



On the other hand, some recent works such as Bitar et al. (2010), Harsha and Dahleh 



(2011), and Kim and Powell (2011) use dynamic programming to address the problem 



of managing the revenue of a renewable generator using storage. In particular, Harsha 



and Dahleh (2011 ) use a stochastic dynamic programming approach to study the optimal 



storage investment problem through characterization of optimal sizing of energy storage 



for efficient integration of renewable resources. In contrast to Harsha and Dahleh (2011), 
we explicitly include ramp constraints in our model, and highlight the effects of ramp 
constraints on the value of storage. Another contribution in this line of research is due to 



Kim and Powell (2011), who study the problem of making advance commitments for a 



wind generator in the presence of storage capacity and conversion losses. Kim and Powell 
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(2011) use dynamic programming to obtain the optimal commitment policy when the 
storage device can have conversion losses but under the assumption of uniformly distributed 
generation from the wind farm. This assumption allows them to derive the stationary 
distribution of the storage level and use it to characterize the economic value of storage. 
In contrast, our results characterize the value of storage purely as an arbitrage mechanism 
that interacts only with the main grid with a guaranteed supply, without any particular 
assumption on price distribution other than the assumption of independent prices. 

The storage problem in this paper also has similarities with the inventory management 
literature in terms of the underlying dynamic programming problem and the corresponding 



optimal policy (for instance, see Federgruen and Zipkin 1986, Kapuscinski and Tayur 



1998, and Goel and Gutierrez 2009). However, the main difference comes from the fact 



that the inventory literature focus on optimally managing inventory in the presence of 
demand, while the model in this paper assumes that the main grid buys all the energy that 
we decide to supply, and instead, focuses on trading inventory under ramp constraints and 
maximizing profit by taking advantage of the fluctuations in the spot prices. Our setup also 
has similarities with the literature on reservoir hydroelectric system management (see, for 



instance, Drouin et al. ||1996 and Lamond, Monroe, and Sobel 1995) especially those with 
price uncertainty. However, since the focus of our paper is on the economic value and price- 
responsiveness of electricity storage as an arbitrage mechanism under ramp constraints, 
the formulation of our model and our results are different from this line of research. 



The model set forth in this paper and some preliminary results were reported in Faghih. 



Roozbehani, and Dahleh (2011). This paper provides a comprehensive exposition which 



adds several new ideas, core analytical results, and systematic computational experiments. 
The contributions of this paper are summarized as follows: First, we propose a dynamic 
model for optimal utilization of storage in the presence of ramp constraints under the 
assumption of independent and exogenous prices. This model assumes limited storage 
capacity and allows sell-back of energy to the grid. Using the principles of stochastic 
dynamic programming, we analytically characterize the optimal policy and the correspond- 
ing value function for the finite-horizon case. In particular, we provide recursive equations 
for computation of the exact value of storage for the finite-horizon storage problem. To ver- 
ify the validity of our assumptions, we apply our finite-horizon optimal policy to real-time 
price data taken from the PJM Interconnection and the Independent System Operator 
(ISO) of New England. We define the competitive ratio (CR) as the ratio of the value 
obtained from our optimal policy to the absolute maximum value that would have been 
obtained deterministically, had we known the entire price process a priori. We then show 
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that the value obtained from our policy under the assumption of independent prices yields 
a relatively high CR when applied to real-world price data, reaching a CR of about 90% in 
some cases. This suggests that the sensitivity of the optimal policy and value of storage to 
the assumption of independent prices is low. We then obtain a closed-form upperbound on 
the infinite-horizon optimal average value per stage of storage over all possible realizations 
of prices within a bounded support. To the best of our knowledge, this paper is the first 
to derive an analytical upperbound on the long-term average value of ramp-constrained 
storage. This result highlights how the capacity limit and the ramp constraint bound the 
value of storage. Next, we show that while the economic value of storage is a non-decreasing 
function of price volatility, the value of storage saturates quickly due to finite ramping 
rates as the capacity increases, regardless of price volatility. Our results on the economic 
value of storage can be used to evaluate certain decisions about investment in storage. 

Next, in order to study the average price elasticity of a storage system in an electricity 
market, we study the average PED in a simulated electricity market over a one-day time- 
horizon, where the term "average PED" reflects the fact that the dependence of storage 
response on the stage and the internal state of storage has been averaged out. We show that 
optimal utilization of storage may, in expectation, induce a considerable amount of price 
elasticity near the average price, but little or no elasticity elsewhere. While the demand 
for electricity has often been considered to be highly inelastic, the existing literature on 
price elasticity are mostly based on empirical evidence and qualitative reasoning, see, for 



instance, Kirschen et al. (2000), Kirschen (2003), Yusta and Dominguez (2002), and 



Faruqui and George (2002). In this paper, we study price elasticity in a quantitative 



framework. To the best of our knowledge, this paper is the first to characterize the PED 
induced by storage through an input-output model of response to prices based on optimal 
control policies in the presence of ramp constraints. Finally, to examine how the storage 
response would have been had we not averaged out the state, we characterize the response 
of storage to exogenous prices and its dependence on the internal state of storage, and 
highlight the interplay between state- dependence and price-dependence of the response 
in a computational framework. To eliminate time-dependence, we study these relations 
in an infinite-horizon setting. We use policy iteration to characterize price responsiveness 
of a storage system over an infinite time-horizon as a function of the storage state, and 
characterize the buy/sell phase transition region in the price-state plane. To highlight an 
interesting implication of the price-state interplay, we study the impact of market-based 
operation of storage on the required reserves, and show that if the ISO does not have 
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perfect information about the exact value of the storage state, the reserves may need to 
be expanded to accommodate market-based operation of storage. 

The remainder of this paper is organized as follows: In Section [2j we introduce the 
dynamic model of utilization of storage. In Section |3j we present the optimal policies for 
the storage management problem and the corresponding value function, and give an eval- 
uation of the performance of the optimal policy. Next, in Section [4] we derive an analytical 
upperbound on the optimal average value per stage of storage, and report our computa- 
tional findings on the economic value of storage. We then discuss the implications of the 
optimal policy in Section [5j by studying the average PED of a storage system in Subsec- 
tion |5.1[ the price responsiveness of a storage system as a function of the storage state 



in Subsection 5.2 and the impact of market-based operation of storage on the required 



reserves in Subsection 15.31 We conclude in Section HI 

2. A Dynamic Model of Storage 

2.1. Notation 

The set of positive real numbers (integers) is denoted by IR+ (Z+), and non-negative real 
numbers (integers) by K+ (Z + ). The probability mass function (PMF) of a random variable 
A is denoted by Pa, and the cumulative distribution function (CDF) is denoted by Fa. We 
will simply use P and F when there is no ambiguity. 

2.2. The Model 

In this section, we develop a dynamic model for optimal management of ramp-constrained 
storage in the presence of stochastically- varying prices. We start by formulating the storage 
management problem as a stochastic dynamic programming problem over a finite horizon. 

The Decisions: The decision set of the storage owner or consumer at each discrete instant 
of time k G Z + is characterized by a pair 

K n , % out )e[o^ in ]x[o,^ lt ] (l) 

where, v ™ and i>£ ut are, respectively, the amount of power that the consumer injects in, 
or withdraws from the storage. The corresponding upper bounds (v m and ^j out ) represent 
the physical ramp constraints on storage. Also, v%. = v™ — v^ ut G [— v out , v m ] denotes the 
net storage response. With a slight abuse of terminology, we may also refer to the storage 
response as demand, with the understanding that v & < implies a negative demand. 
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The Price: The price at each stage X k is sampled from an exogenous stochastic process 
that is independently distributed across time, with mean A& and standard deviation <t & , 
and support over [A™ 111 , A™ ax ] C [0, oo). It is assumed that at the beginning of each time 
interval [k, k + 1] , the random variable A& is materialized and revealed to the consumer. 
Note that the distributions of A& can be different at different stages; however, we assume 
that the price distribution at each stage is known a priori. We also assume that the feed-in 
and usage tariffs are the same, i.e., Afc is the price per unit for both purchase (corresponding 
to Vk > 0) and sell-back (corresponding to v k < 0), and there are no transaction costs. 

The States: The storage state is characterized by a variable 

s k e[0,s] (2) 

where Sk is the amount of energy stored, and s is the upper bound on storage capacity. 
The state Sk evolves according to: 

s k+1 = $s k + r/X 11 - rt ut (3) 

where (3 < 1 is the decay factor, r/ m < 1 and ?7 out > 1 are charging and discharging efficiency 
factors. Note that efficiency factors and ramp rates might in general be complicated func- 
tions of the operating point, i.e., the storage level, but in this work we focus our attention 
on an ideal case. The idealized model of the dynamics of storage can be written as: 

s k+ i = s k + v k , v k e [-v ont , v in ] (4) 

which corresponds to (3 = 1, rj m = 1, and ?7 out = 1. 

Penalty and Salvage Value: There is a penalty h k (s k ) associated with storage, where 
the sequence of functions hk : M+ h-> R + are assumed to be monotonic. Also, a salvage 
value of t e [A™ n , A™ ax ] is assigned to each unit of energy left in storage by the end of the 
time-horizon. 

The Optimization-Based Model of Ideal Storage: Since our goal in this paper is to 
develop tractable models (and derive bounds) that effectively highlight the important 
structural features of the optimal control law and the associated economic value of storage 
with an emphasis on the ramp constraint and storage capacity, we will adopt the idealized 
model of storage. Nevertheless, in terms of methodology, it would be straightforward to 
include a "price-adjustment" factor in our formulation to account for injection-withdrawal 



losses in a similar manner done in Secomandi (2010). In addition, the piecewise-linear 



penalty function that we have embedded into our model can be used as a surrogate cost 
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to model the dissipation losses associated with keeping energy in storage. Note also that 



according to Qian et al. (2011), the academic state of the art is around 95% efficiency 



for a battery pack and 93% for the overall system with converters. The industrial state of 



the art is around 90% efficiency for batteries (A123 Systems 2012). We also assume that 
the ramp constraint is symmetric, i.e. v m = v out = v. The idealized storage management 
problem can be formulated as a finite-horizon dynamic programming problem as follows: 

min E [2^ k=0 + h^k} ~ *sjvJ (5) 

S.t. S k+1 = s k + v k 
s k G [0,s] 
v k E [-v, v] 

X k exogenous, and independently 
distributed according to a PMF P k 

Remark 1 . We first formulate and solve the storage problem for the finite- horizon case. 



Later in Section 4.1, we consider the infinite- horizon case and obtain an upperbound on 



the optimal average value per stage of storage. 

3. Characterization of the Optimal Policy and an Evaluation of Its 

Performance 
3.1. The Optimal Policy 

In this subsection, we characterize the optimal policy and the value function for problem 
([5| based on principles of stochastic dynamic programming. 

Definition 1. Given a sequence of probability mass functions P k , k = 0, 1, . . . , N, let <d k 

and ipk be sequences of maps from the set of all subsets of E + to K + , defined as follows: 



e k :i^J2 op k(Q)i V/Cl + (6) 

^ fc :J^^P fe (0), VJCR+. (7) 
eel 

Given v E M+, and maps and i\) k as defined in (|6j) and ((7|, is the map from the set 
of all subsets of M + to K defined according to 

$l:I^v(Q k -fnj; k )I, V/CR+, (8) 

where 

p = inf /. 

For instance, $^ maps an interval (a, b) to (0^ — aipk) (a, b) . 
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Theorem 1. Consider the finite-horizon storage management problem |5p with s = nv 
for some n G Z+. Furthermore, assume that the penalty functions hf. : [0, oo) — > [0, oo), 
k = 0, . . . , N are piecewise linear non- decreasing convex functions of the form: 



hk (s) = h l k s + c\, s G [iv, (i + l)v) , i G Z + 



Then, 



(i) the optimal policy is characterized as follows: 
if Sk G [iv, (i + l)v), for some i G Z + , then 

' max(— Sk, —v) , t™^ ),% l ' < Afc for all i 
iv-Sk , t\ +l <X k < fori>\ 

(i + l)v - s k , 4+1 < A fc < t{ +1 for all i 
yv , Afc < t 1 ^ for all i 

where the thresholds are computed via the following recursive equations: 



(9) 



(10) 



t 



N 



t 

-h 



iV) 



zG{0,l,2,...,n-l} 

i > n 



for k<N : 

^A; = _ ^fc + ^&(^fc+l)*fe+l] + (^fc+1 — tk+l)Fk (*fe+l) 



1>1 



(11) 



fzij the value function is a piecewise linear convex function of the form: 

V k (s) = -t\s + e\, s G [iv, (i + l)v) , i G Z + 



(12) 



where t\ are the thresholds given in ( 11 ) and the intercepts e\ are computed via the following 
recursive equations: 



0, 



i6{0, 1,2, ...,n-l} 



s(t l N — t), i>n 



for k<N : 

e°k = c° k + e° k+1 + (v\f« + 4 +1 - e° fc+1 - vt\ +l )F k {tl +1 ) 

e fc = c fc+ t,/ ^ + /(*fc+i)^l+i)*fe+i) e fe+i' e fe+iJ e fe+i) +fi , (*l+i^l+i) ^fe+i)) * — 1 



(13) 



where the functions f and g are given by 
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/(•) = 4-\ - vti\\ + (4 +1 - el- + \)F k (tl + \) + (e£\ - e* +1 )F fc (4 +1 ) 
Proof. Please see the Appendix. □ 

Figure [I] shows how the thresholds vary with time and state for the case of a discretized 
truncated log-normal distribution with mean X k = 49 and a k = 9 for all k, i.e. for inde- 
pendently and identically distributed (i.i.d) prices. For generating this plot, we set v = l, 
n = 15, N = 24, t = Afc, and assume no storage penalties (i.e. h\ = for i < n and all k). 




Figure 1 Thresholds as a function of time and state for a discretized truncated log-normal distribution (and 
i.i.d prices) with mean Afc = 49 and — 9 for all k, v = 1, n=15, N = 24, i — \k and no storage penalties. 

Remark 2. The form of the optimal policy shows that if we start with an empty storage 
(sq = 0), then the storage state s k will only take integer multiples of v since v* k G {— v, 0,v} 
for all k. When Sq 7^ 0, the storage state will fall on the grid of integer multiples of v 
immediately after the first time that = iv — s k or v^.= (i + l)v — s k , and we will have 
v* k G {— v, 0,v} for the remainder of the time horizon. This conclusion holds for the infinite- 
horizon case as well and can help simplifying the policy and analysis by focusing on a finite 
state system. 

Remark 3. The upperbound s on the storage capacity is enforced by choosing h\ in 
([9]) sufficiently large (i.e. h\ > A™ ax ) for i > n, so that it would never be optimal to store 
energy beyond s. It can be verified that the thresholds and consequently the optimal policy 
are invariant with respect to the choice of h\ for i > n as long as h\ > A™ ax . 

Definition 2. We define the economic value of storage, or simply the value of storage, as 
the negative of the cost of the optimal value of problem ([5| divided by the number of stages 
(iV), and we denote it by V for the finite- horizon case and by Vqo for the infinite- horizon 
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case. Therefore, V = —Vq (sq) /N . For instance, if so = (i.e. the consumer starts with an 



empty storage), it then follows from (12) that for the finite- horizon case: 



V = -V o (0) = -e° /N. (14) 



Thus, V can be computed using the recursive equations in (13) for the finite-horizon case. 



3.2. The finite-horizon value of storage under empirical price distribution from 
real-time wholesale electricity markets 

Herein, to examine our results and verify the validity of our assumptions, we test the 

optimal policy against actual market price data. We apply the finite-horizon optimal policy 



(10) to real-time wholesale market data taken from the PJM Interconnection (2011) and 



the ISO New England (2011) to examine the competitive ratio (CR) of our policy when 
applied to real-world price data. This would allow us to assess whether the assumption of 
independent prices is actually reasonable for the storage problem. A relatively high CR 
would not necessarily suggest that prices are independent; rather, it could mean that the 
sensitivity of the optimal policy to the correlations, if any, in real-world prices is low. If 
the CR obtained form the assumption of independent prices is relatively high, we can 
suggest that the additional information that the current price provides on future prices (as 
a conditional distribution) has little value. 

The setup of the experiment is as follows. We take the actual data for hourly energy 
prices (for 16 hours of each day, from 8 a.m. to 12 a.m.) for two different months (December 



2010 and July 2011) from the PJM Interconnection (2011), and May and November 2011 



from ISO New England (2011 ). The choice of these dates and times was arbitrary. For the 
purpose of these simulations, we set t, the salvage value, equal to A, the empirical mean. 
To perform the simulations, we first find the empirical distribution of the data for each 
case. In these computations, we use the same price distribution for all k; i.e., we take all 
the price data for the entire month, and use this data to find the empirical distribution of 
prices in that month and compute the thresholds for the optimal policy for all hours from 
the same empirical distribution. In other words, we assume that prices are independently 
and identically distributed (i.i.d). Then, at the beginning of each stage we reveal the actual 
price of that stage to the optimal policy and record the decision. This gives us the profit 
resulting from the optimal policy. Next, for the purpose of comparison, we assume the 
entire price sequence is perfectly known a priori and compute the value that results from 
deterministically and omnisciently maximizing the profit against the materialized prices. 
This deterministic value is the absolute best that an omniscient agent could have done, 
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Table 1 Competitive Ratio for each set of simulations 



Source/ Month 


Competitive Ratio 


# of qualifying 


; days Range from the mean 


PJM / July 


0.86 


27 


1 x a 




0.84 


28 


1.5xa 




0.81 


All 


All 


PJM / December 


0.88 


22 




0.87 


25 


1.5 x a 




0.77 


All 


All 


ISO NE/ May 


0.72 


25 


lxcr 




0.71 


All 


1.5 x a 




0.71 


All 


All 


ISO NE/ November 


0.89 


29 


lxcr 


0.88 


All 


1.5 x a 




0.88 


All 


All 



and we will refer to its corresponding deterministic policy as the omniscient policy. We 
then find the CR by computing the ratio of the value obtained from the optimal policy to 
the value obtained from the omniscient policy. The CR gives us a measure of how well our 
optimal policy has performed. 

For each month, there is a number of days in which the average price is well above the 
average price of the entire month. These days are outliers in the sense that the empirical 
distribution is way off for modeling their price sequence. We go about this issue by per- 
forming three sets of simulations. For the first set of simulations, we only select those days 
in which the average price is within one standard deviation of the average price of all the 
selected days in the month. In the second set of simulations, we select those days in which 
the average price is within about 1.5 times the standard deviation of the average price of 
all the selected days in the month. Then, in the third set of simulations, we take all the 
days of the month into account, even the outliers. We record the CR for each day, and 
report the average CR for each month and each case in Table [1} Comparing the results in 
Table [I] reveals how much these outlier days affect the CR. Note that a ramp constraint 
of v = 1 and a storage capacity of s = 10 is used in all the simulations, and it is assumed 
that there are no penalties on storage. 

Figure [2] shows the plots of the empirical value of storage for each day of the month in 
the second set of simulations (i.e for those days whose average price is within 1.5 times the 
standard deviation of the average price of all the selected days in the month). 

As it can be seen in Figure |2j we have nearly perfect matching in some days, while in 
some other days there is discrepancy. This discrepancy is mainly due to two reasons. First, 
it is because we have multiple spikes in some days with only a few (or no) prices that are 
below our buying thresholds. So, even though there are ample opportunities for arbitrage, 
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Figure 2 The empirical value of storage for each day of the month in the second set of simulations, obtained 
from applying the optimal policy (solid line) and the omniscient policy (dashed line) 

even the lowest of these spiky prices is not within the normal range. In Table [TJ this effect 
can be observed for the month of December in PJM, in which the removal of those days 
with high average prices improved the CR from 0.77 to 0.88. The second reason is that in 
some days all the prices are almost in the same range compared to our thresholds (i.e. they 
are either mostly above our buying thresholds or mostly below our selling thresholds). In 
other words, the low CR is just an outcome of an undesirable sequence of prices (sample 
path). So, even though for the deterministic case with the price sequence known a priori 
it is possible to take advantage of these small price differentials, our thresholds are unable 
to capture these opportunities. This effect can be observed in the month of May in Table 
[TJ in which the removal of those days with high average prices did not really improve the 
CR; this is because the average price in the days with a relatively flat price sequence is 
not necessarily considerably higher than the month's average. 

So far, we have been computing the thresholds for the optimal policy using the empirical 
price distributions from the prices in that same month. This can be taken as a proxy for 
the sensitivity of the value of storage to correlations in the actual prices, given that our 
optimal policy assumes independent prices. Although there is no benchmark to compare 
with, the CR obtained from applying the optimal policy seems reasonably high, and we 
suggest that the sensitivity of the value of storage to the assumption of independent prices 
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is low. Also, although we do not do as well on those days with higher than normal, or very 
flat price profiles, it does not appear that a Markovian or Martingale assumption on prices 
could do any better, because these prices are really outliers and do not seem to follow a 
structured stochastic pattern that can be learned from past data. However, this needs to 
be substantiated by further studies and systemic experiments. 

In the next set of experiments, instead of computing the empirical distribution from the 
data of that same month, we compute the empirical distributions from the price data of 
the past 30 days, the past 20 days, and the past 10 days, respectively. The results are 
shown in Tabled 

Table 2 Competitive ratios using the empirical distribution from the past 30, 20, and 10 days 



Source / Month 


Past 30 days 


Past 20 days 


Past 10 days 


# of selected days 


PJM / July 


0.68 


0.70 


0.75 


All 


PJM / December 


0.55 


0.65 


0.67 


All 


ISO NE/ May 


0.69 


0.70 


0.68 


All 


ISO NE/ November 


0.91 


0.89 


0.89 


All 



An interesting observation is that for both months of May and November in ISO NE, 
using the empirical distribution from the past 20 days gives almost the same CR as using 
the empirical distribution from May and November themselves (as reported in Table [TJ . 
However, comparing the competitive ratios shown in Table [2] for both months in PJM with 
the results shown in Table [TJ we observe that for PJM, using the empirical distribution 
from historical data does not do as well as using the empirical distribution from that month 
itself. 

4. Characterization of the Economic Value of Storage 
4.1. Analytical upperbound on the value of storage 

In this subsection, we will derive a bound on the long-term economic value of ramp- 
constrained storage. Herein, for the purpose of obtaining the bound, we further assume 
that h\ = 0, for all i < n — 1 and k < N. This assumption is consistent with our objective 
of finding an upperbound on the value of storage. 

Definition 3. Given a control policy TTk '■ [0, oo) 2 i— > [—v,v], and starting from an arbitrary 
initial state s, the infinite-horizon average cost per stage associated with problem ([5j) is 
defined as 

(15) 



def 1 „ 

7„. = hm — E 



'N-l 
.k=0 
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where Vk = ^k(xk, A&). We will refer to the problem of optimization of 7^ over all feasible 
stationary policies as the infinite-horizon storage management problem. We will denote the 
associated optimal cost by 7* and refer to Vqo == —7* as the long-term expected economic 
value of storage. 

Remark 4. It is standard to show that the optimal average cost is independent of the 
initial state Sq. Moreover, if the relative value iteration for the infinite-horizon storage 
problem converges to some differential cost function H*(s), then it is necessary for H*(s) 
and the optimal average cost per stage 7* to satisfy the Bellman equation (see, for instance, 



Bertsekas 2000): 



H*(s) = E 



mm 

u£[max(— s,— v),min(v,s— s) 



Xv + H*(s + v) 



-7 



(16) 



Theorem 2. Consider the infinite-horizon storage management problem. Suppose that 
the support of the price distribution function at all stages lies within an interval 
[A m in, A max ] C [0,oo). All else held constant, the maximum over all possible distributions, of 
the long-term economic value of storage is given by 

v(X 

max A mm 

) n (A 1: 



-7 



"max A . 1 1 j . • ) S 



(17) 



2 n + 1 2 s + v 

and is attained when the prices are sampled from a two-point symmetric distribution with 
nonzero probability masses placed at the endpoints of the fixed support: 



1/2 if A = A 
P v {A!=<;i/2 if A = A 

otherwise 



mm 

max 



(18) 



Furthermore, the corresponding differential cost function satisfying the Bellman equation 
is given by the following piecewise linear convex function: 

(i + l)X min +(n-i)X 

1 



H*(s) 



'm ix ' (' I ) (A : ,i, iN A . . , j . • ) r 

-s 



(19) 



n + 1 2(n + l) 

where s G [iv, (i + l)v) and i e {0, • ■ ■ , n — 1}, and for the special case of s = s, we take 
i = n — l. 

Finally, for any two-point distribution with PMF 

{a if A = A max 

l-o */ A = A min (20) 
otherwise 

we have 



* _/a . n 6(l + 6 + --- + 6 n " 1 ) 
V 00 = - 7 =v(X max - X min ) {b + m + b+ _ _ + fcn) (21) 



where b = (1 — a) /a. 
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Proof. Please see the Appendix. □ 

If in addition to the support of the price distribution we also fix the mean of the distri- 
bution, we can obtain a tighter bound as stated in Corollary [l] 

Corollary 1. Suppose that in addition to fixing the support of the price distribution 
function, we also fix the mean of the price distribution to fx G (A min ,A max ). All else held 
constant, the maximum over all possible distributions, of the long-term expected value of 
storage is attained when the prices are sampled from a two-point distribution with the 
following PMF: 

{ M-^min Af \ — \ 

\ _\ . V " A max 

''max A min 
A max A min 
otherwise 

The corresponding long-term value of storage is obtained by plugging b = (A max — (j,)/(p — 



A m in) into (21) 



Proof. Please see the Appendix. □ 

Remark 5. The n/(n + l) scaling in the optimal average cost per stage implies that yu/o 
of the maximum possible value of storage is achieved when the storage capacity is only 9 
times the ramp constraint. Note that this was obtained for an extreme distribution. As we 
will see in the remainder of this section, for less extreme distributions with smaller variance 
over the support, the value of storage saturates even more quickly. This includes empirical 
distributions obtained from electricity market data. Furthermore, aging, dissipation, and 
non-ideal charging and discharging factors further reduce the value of storage. 

4.2. Computational Experiments for Characterization of the Economic Value of 
Storage 

In this subsection, we employ numerical computations to characterize the economic value 
of the proposed model of storage over a finite time-horizon and highlight the effects of 
ramp constraints and price volatility on the value of storage. 

Herein, we consider the following classes of distributions: 

• Discretized truncated log-normal distribution, with fixed mean A = 50, 

• Discretized uniform distribution, with fixed mean A = 50. 

The reason for choosing the log-normal distribution is that the empirical distributions from 
ISO New England and PJM qualitatively resemble a log-normal distribution, at least for 
the cases that we tested. The choice of the mean (A = 50) is also a realistic choice for 
average hourly energy prices in markets such as PJM and ISO NE. For the purpose of 
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Figure 3 V vs. n (left) and a (right) for a few samples, using a discretized truncated log-normal distribution. 

these computations, we use the same price distribution for all k (i.e. we assume that prices 
are independently and identically distributed). Throughout this section we assume that 
So = 0, which means that the consumer starts with an empty storage. For each of these 
distributions, we fix all quantities in our model other than a, the standard deviation of 
price distribution, and n, the ratio of storage capacity s to physical ramp constraint of 
storage v. We vary n — s/v by fixing v and changing s. Using the fixed quantities iV = 24, 
v = 10, and A = 50, we examine how V varies as a function of a and n for the case of no 
storage penalties. For the purpose of these simulations, we set t equal to the mean of the 
price distribution. Herein, we set h\ = 0, for all i < n — 1 and k < N, so that there is no 
penalty on storing energy up to capacity. Then, for a fixed time horizon, we examine how 
V varies with a and n, for each of the following price distributions: 

Discretized truncated log-normal distribution: Figure [3] illustrates how V changes with a 
and n, for the discretized truncated log-normal distribution. The plots show that the value 
of storage increases linearly with a. As one would expect, the value of storage also increases 
as the storage capacity increases. However, it is interesting to note that for a fixed standard 
deviation, the value of storage saturates fairly quickly as a function of n. Hence, for a 
given time horizon, a fixed ramp constraint, and a fixed cr, there exists a certain range 
for capacity beyond which the value of storage will no longer change noticeably. Also, the 
optimal storage capacity increases with price volatility. 

Discretized uniform distribution: As can be seen in Figure |4j saturation of the value of 
storage occurs at about the same value as in the log-normal case. However, for certain 
extreme distributions, such as an asymmetric 2-point distribution, saturation can occur 
more quickly. Note also that the value of storage is a linear function of the standard 
deviation, just like the log-normal case. 

One interesting observation in these results is that in the presence of ramp constraints, 
several distributed storage systems would be more profitable than one large storage system 
of equal ramp constraint and aggregate capacity, due to the quick saturation of V as n 
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Figure 4 V vs. n (left) and a (right) for a few samples, using a discretized uniform distribution. 

increases. Although this observation is based on the assumption that the ramp constraint 
and capacity are independent, this assumption might actually be valid for the case of 
distribution grids in which the ramp constraint is imposed by the power lines. Another 
interesting observation is that although the shape of the plots look quite similar for both 
distributions, the value of the uniform distribution is higher than that of the log-normal. 
This makes intuitive sense because with the uniform distribution, we have as many oppor- 
tunities for buying at a low price as we have for selling at a high price, while for the 
log-normal case, most of the probability mass is centered around the mode, which creates 
fewer opportunities for arbitrage. 

Using wholesale market data from ISO New England: For the purpose of comparison, 
we repeat the experiment using the data for hourly energy prices from three different 



months (April, June, and October of 2011) obtained from ISO New England (2011). For 
each month, the hourly data (from all 24 hours of all days of the month) have been used 
to find the empirical distribution of prices for that particular month. For the purpose of 
these computations, we compute the empirical distribution over all stages, and we use this 
empirical distribution for all the stages (i.e. we assume that prices are i.i.d). Figure [5] shows 
how V varies with n in each month without storage penalties: 




*— ISO NE in April (li=44.03, o=15.24) 
o ISO NE in June (ii=43.46, o=16.43) 
b - ISO NE in October (li=39.60, o=12.21) 



Figure 5 V vs. n for 3 different months, without storage penalties 

Note in Figure [5] that similar to the results of the distributions used in Section 4.2, the 
value of storage nearly saturates when n is about 6. An interesting observation is that 
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although both the mean and the standard deviation of the empirical price distribution in 
October are lower than those of the empirical price distribution in April, the value of storage 
is higher in October. This observation reiterates the importance of the price distribution 
for the value of storage, even when its mean and standard deviation are somewhat lower, 
though many other factors such as non-stationarity and correlation in the data could skew 
the results. 

5. Implications of the Optimal Policy 

Herein, we will focus on understanding the price-dependence and state-dependence of the 
optimal policy, and the impact of the state-price interplay on the storage response. We 
first average out state- dependence of storage response and focus our attention on "aver- 
age" price-responsiveness only, which allows us to address the expected price elasticity of 
demand induced by storage in the first subsection. Then, in the next two subsections, we 
focus on the interplay between state-dependence and price-dependence of storage response, 
and study potential impacts of the state-and-price-dependent response from market-based 
storage on the required reserves. 

5.1. Average Price Elasticity of Demand Induced by Storage 

In this subsection, in order to study the implications of the optimal policy on price elas- 
ticity of demand (PED) in an electricity market, we present a computational framework 
for studying the average PED in a simulated electricity market. In our dynamic model, 
the storage response depends on price, stage, state, time-horizon, storage capacity, and 
ramp constraint. The term "average PED" reflects the fact that the dependence of stor- 
age response on the stage and the internal state of storage have been averaged out. We 
assume that there is a fixed time horizon N, and eliminate state- dependence by taking 
expectations. In particular, we define: 

v(k,X)=E S(hSk [v* k \X k = X]. 

In order to eliminate stage-dependence, we think of the storage response-measuring 
observer as sampling a random time r uniformly over {0, • • • ,N}. By averaging over this 
randomness, we maintain dependence on price alone: 

v avg (X)=E T [v(r,X)\, 

which is captured in the simulations by clustering real-time prices, and averaging over each 
cluster. 

In these numerical simulations, we average over random instances of price and storage 
initial states. We set N = 288, which corresponds to a period of 24 hours, where real-time 
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prices are updated once in every 5 minutes. The storage system implements the opti- 
mal policy given in Theorem [T] For the purpose of these computations, we use the same 
price distribution for all k (i.e. we assume that prices are independently and identically 
distributed). For generating random price sequences, we simulate a discretized truncated 
log- normal distribution with A = 52 and a = 22, because the log-normal distribution qual- 
itatively resembles the empirical distribution of prices from ISO NE and PJM, at least 
for the cases that we tested. Also, a mean of 52 and standard deviation of 22 are realistic 
choices for real-time energy prices in markets such as PJM and ISO NE on a day with 



moderate volatility. Based on our results in Section ^2, for these model parameters, a 
storage capacity of s = 5v is a reasonable choice for all consumers. We set t equal to A in 
these simulations, just like all previous simulations. We also set v = 10, and h\ = for all 
k < N and i <n. 

Figure [6] illustrates how the average storage response changes as a function of price. 
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Figure 6 Average storage response vs. price, using the discretized truncated log-normal price distribution. 

As the plot suggests, the expected demand seems to be considerably more responsive to 
changes in the prices that fall in the mid-portion of the price range. This portion serves 
as a steep transition region, in which the policy quickly switches from the "buy" policy to 
the "sell" policy. 

To characterize price elasticity, let us first recall the standard definition of PED: 

< 22 > 

where d denotes demand. To characterize PED more accurately, one needs to bear in mind 
that the overall PED should have the firm component of demand in it. Hence, in this setup, 
we set d = df + v avg (X), where d* denotes the firm component of demand. We can observe 
in Figure [6] that the average PED is almost zero for prices that are considerably larger 
or smaller than the mean price, and only in the mid-portion of the plot (i.e. around the 



mean price) we notice a substantial average PED. One can verify using equation (22) with 
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Table 3 Average price elasticity of demand around the mean price using different values for d f 

~cP Average Price Elasticity 



v 

Sv 
8v 



-3.6 
-1.2 
-0.45 



d = dJ + v avg (X) that the average PED (i.e. the PED computed using the average storage 
response) depends on how the average storage response compares with the firm demand. 
Table shows the average PED around the mean price, using different values for d? . 



5.2. Price Responsiveness of a Storage System 

In this subsection, we present a computational framework for understanding the behavior 
of storage as a function of price and the amount of stored energy, and for characterization 
of the buy/sell phase transition region in the price-state plane. In order to eliminate stage 
dependence, we will consider the infinite-horizon version of the storage problem ^ and 



perform policy iteration (see, e.g., Bertsekas 2000) to numerically obtain a stationary 
(stage-independent) policy for purchase/sale as a function of both state and price. This 
section provides a qualitative picture of the structural characteristics of the behavior of 
storage, and a framework for estimating the PED as a function of the state. Herein, we 
will assume that the consumer starts with an empty storage, implying that the states 
would only take on integer multiples of the ramp constraint. We also use the same price 
distribution for all stages (i.e. we assume that prices are i.i.d). In our computations, we 
first use a discretized truncated log-normal price distribution, and then compare the results 
against the case of a discretized uniform distribution. For both distributions, we use a mean 
of about A = 50 and a standard deviation of about a = 30, and also the same support. We 
set v = 1 and n = 10. Figure [7] illustrates how the storage response varies with price for 
three cases of the state (when the storage is empty (i = 0), when the storage is half full 
(i = n/2), and when the storage is nearly full (i = n — 1)). 
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Figure 7 Storage response vs. price for three sample states for discretized truncated log-normal (left) and 
discretized uniform (right) price distributions, both with mean 50 
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Note in Figure [7] the considerable effect of the storage state on storage response, com- 
pared to the small effect of the price distribution. More specifically, for both distributions, 
when the storage is empty, the optimal policy recommends purchasing from the grid even 
when the prices are somewhat above the mean price. Note that for the log-normal case, 
this policy change occurs at a slightly lower price because of the left skewness of the log- 
normal distribution. Though, when the storage is half full, we switch from the "buy-it-all" 
policy to the "sell-it-all policy" right at the mean price for the uniform distribution; for the 
log-normal distribution, this policy change occurs slightly before the mean price, which is 
again due to the left skewness of the log-normal distribution. Finally, when the storage is 
nearly full, for both distributions the optimal policy is to sell as much energy as possible 
for most prices, and to do nothing for the low prices. 

The transition points in the infinite-horizon policy from sell-it-all to buy-it-all on the 
s — A plane are shown in Figure [8j Any point to the left of and / or below the transition 
points is a buying policy, which corresponds to v*(s, A) = v, and any point to the right of 
and/or above the transition points is a selling policy, which corresponds to v*(s,X) = —v. 
The plus signs show a direct transition from buying to selling when moving along the 
vertical axis, i.e., as storage state varies, unless they are immediately followed by a star 
on their right. The stars denote a transition through a "Do Nothing" policy when moving 
along the horizontal axis, i.e., as price varies. Therefore, at the prices denoted by * we 
have v* = 0. Figure [8] clearly illustrates the interplay between the state-dependence and 
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Figure 8 Occurrence points of policy change from sell-it-all to buy-it-all for discretized truncated log-normal 

(left) and discretized uniform (right) price distributions 



the price-dependence of the storage response. Note also in Figure [8] the small effect of the 
price distribution on the storage response. The optimal policy for the log-normal case is 
slightly more shifted to left compared to that of the uniform case, which is again caused 
by the left skewness of the log-normal distribution. 
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We can now use the above results to characterize the PED as a function of the storage 



state. In order to compute the overall PED using (22), once again we quantify the demand 
(d) such that it includes the firm component of the demand as well: 

d=d f + v*(s,X). (23) 

We have PED(s) = for all the points in the "Buy Region" and in the "Sell Region" 
because the storage response is constant in those regions. However, around the transition 
curve, the PED is non-zero because Ad < in that region. But Ad can only take two 
values: Ad = —2v when there is a direct transition from the "Buy Region" to the "Sell 
Region" , and Ad = —v when the transition is through a "Do Nothing" policy. Hence, we 
have 

PED(s) = -— r 2vX and PED(s) '" A 



(df + v*(s,\))A\ v ' (df + v*(s,\))A\ 

around the points denoted by + and *, respectively, where, depending on the point at 
which we choose to compute PED, v* takes on one of the values in {— v, 0,v}. 

5.3. Impact of Market-Based Operation of Storage on the Required Reserves 

In this subsection, we will evaluate how the optimal response from storage affects the 
amount of reserves (both generation and demand) required to guarantee that supply and 
demand can be matched. We assume that the Independent System Operator (ISO) pri- 
marily uses renewable generation with zero marginal cost and a conventional generation 
source with a quadratic cost to meet the demand. It is assumed that the overall demand 
consists of the storage response and a deterministic demand with both elastic and inelas- 
tic components. In our simulated setup, at the beginning of each pricing period, the ISO 
predicts the amount of renewable generation available during that period, possibly with 
some error. We assume that the amount of renewable generation in each period is i.i.d 
and is sampled from the bimodal distribution shown in Figure [9| which can for instance, 
correspond to a system with two renewable sources, one with high capacity and one with 
low capacity. In the event that during any time period, there is a shortfall in renewable 
generation compared to what was predicted, the ISO extracts from the generation reserves. 
Similarly, if there is an excess of renewable generation, the ISO would direct the excess 
generation to the demand reserves. We assume that the consumers can learn only the 
stationary distribution of prices (under the assumption of i.i.d prices), and not the exact 
mechanism by which prices are generated, and that the consumers are rational, and hence, 
they have no incentive to manipulate their storage response. Thus, the storage manage- 



ment policy discussed in Section |5.2| is deemed optimal by the consumers. The setup is 
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Figure 9 Matching demand and supply by the ISO 



shown in Figure [9j In this setup, at the beginning of each time period, the ISO determines 
the clearing price by solving the following equation for A&: 



a\ k + w k = v*(s, A fc ) + dfc(Afc) 



(24) 



where is the predicted amount of renewable generation during the k-th time period, 
t>*(s,Afc) is the optimal response of storage obtained using policy iteration as discussed 



in Section 5.2 dk(Xk) is the deterministic demand, and a\k is the optimal response of a 
supplier with quadratic cost c(x) = ^x 2 , to a given price A^. For the purpose of numerical 
computations, we assume that dk(Xk) is a logistic function (as shown in Figure [9]) 

(30 MWh)e" Afc / 30 



4(A fc ) = 5MWh + 



(25) 



e -A fe /30 + e -A/30 

The choice of a logistic function for characterizing the deterministic demand was due to 
the fact that qualitatively similar functions have been used in the past for modeling price 
responsive demand in electricity markets (see, e.g., Carrion, Conejo, and Arroyo |2007 ), this 
choice is otherwise arbitrary and not central to our analysis or conclusions. The choice of 
the numerical values of the parameters of the logistic function is reasonable for the hourly 
load in a small power system. For all k we set s/v = n = 5, the average price A = 50 $ /MWh, 
and vary the ramp constraint for experimentation, looking at three cases: v = 0.25 MW, 
v = 0.5 MW, and v = 1 MW such that the storage capacity s is roughly 4.17%, 8.33%, and 
16.67%, respectively, of the maximum possible demand in the absence of storage. We also 
set a = 4 (MWh) 2 /$ so that the average price in the simulated setup is about 50 $/MWh, 
which is close to the typical average hourly prices in various electricity markets. Note that 
since we are fixing n, all the storage devices across the grid become synchronized in the 
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Table 4 Change in the required reserves for various values of ramp and reliability levels 
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steady state, and hence, we can use one large value for v which serves as a proxy for the 
cumulative ramp rate constraint across the grid. 

We further assume that the prediction of the amount of renewable generation available 
in each period has an error that is sampled from a truncated normal distribution with zero 
mean and standard deviation 0.5 MWh, and that the ISO estimates the storage state with 
an error that has a discrete uniform distribution over the support {— v, 0,v}. We compute 
the percentage change in the amount of reserves needed in the presence of storage compared 
to the case of no storage, for each storage capacity mentioned above. In each scenario, we 
first examine the 100% reliability level (i.e. when demand and supply are guaranteed to 
match roughly 100% of the time) and then we repeat the computations for 99% and 98% 
reliability levels. 

As the simulation results in Table [4] suggest, market based operation of storage may 
require an increase in the amount of required reserves (both demand and supply reserves) . 
The larger the amount of energy that can be stored, or extracted from storage, the higher 
the amount of demand and supply reserves needed. Note that in the largest case of storage 
reported in Table |4j the storage capacity is only one sixth of the maximum demand, while 
the reserves need to be expanded by about 40% to accommodate the integration of this 
much storage (the histograms of the required reserves for this case of storage capacity are 
shown in Figure [9]). 

This effect is mainly caused by the threshold-based, state dependent response of storage 



as discussed in Section 5.2 (Figure [8]). When the prices are high due to lack of renewable 
generation, the storage system does not necessarily respond to the price incentive the way 
the ISO would want it to, because even if the prices are above the average, the optimal 
policy for the storage could be to buy if the storage level is low. Hence, in the event that 
the actual amount of renewable generation falls short of what was predicted, if the ISO 
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also makes an error in estimating the storage state, a considerable amount of energy may 
be extracted from the grid by the storage device, which would force the ISO to use the 
generation reserves even more. Analogously, if there is an excess of renewable generation, 
the ISO may be forced to use the demand reserves even more if the storage device feeds 
energy back into the grid. A higher penetration of storage, and hence, a higher aggregate 
ramp rate amplifies this issue because the impact of erroneous state estimations would be 
even more profound. However, the generation reserves are typically fast generators with 
high economic and environmental costs. Hence, the ISO may need to regulate market-based 
operation of storage devices to prevent such impacts, and/or design mechanisms for pricing 
these externalities. Also, the ISO may need to design mechanisms that guarantee its access 
to the exact value of the storage state, because the intense interplay between state and 
price is such that estimating the storage state even with a small error can heavily impact 
the storage response. 

6. Conclusion 

In this paper, we proposed a dynamic model for optimal control of storage under ramp 
constraints and exogenous, stochastic prices. We derived the associated optimal policy and 
value function, and gave explicit formulas for their computation. Moreover, we derived an 
analytical upperbound on the long-term average economic value of storage, which is valid 
for any achievable realization of prices over a fixed support, and highlights the dependence 
of the value on ramp constraints and capacity. This result can be useful in assessing viability 
of investment in electricity storage. We also showed that while the value of storage is a 
non-decreasing function of price volatility, due to finite ramping rates, the value of storage 
saturates quickly as capacity increases, regardless of price volatility. We highlighted the 
dependence of the response of storage to prices on the internal state of storage, and also, by 
averaging out state and stage dependence, we showed that in expectation, storage induces 
a considerable amount of price elasticity near the average price. We also showed that if 
the ISO does not have perfect information about the exact value of the storage state, the 
reserves may need to be expanded to accommodate market-based operation of storage. 

Our results provide insight into learning the behavior of storage, particularly modeling 
and estimating the response of a ramp-constrained storage system when used as an arbi- 
trage mechanism. We used price data from real-time wholesale markets to examine the 
sensitivity of the optimal policy and the value of storage to our assumption of indepen- 
dent prices. The relatively high competitive ratios that we found when testing the optimal 
policy suggest that the sensitivity of the policy to the assumption of independent prices is 
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low. The competitive ratios found in this paper by applying the optimal policy can be used 
as a benchmark for comparison with other results that assume more complicated models 
of the stochastic price process. 

We did not directly consider the effects of inefficiencies such as conversion losses and 
aging on the optimal policy. While such inefficiencies certainly limit the economic value 
of storage, an important question to ask is what is their effect on the response of storage 
to prices? For the case of battery storage, aging is proportional to the amount of current 
withdrawn or injected. Intuitively, this would make buying energy for, or selling energy 
from storage less profitable at moderate prices. Within the class of threshold policies, the 
corresponding optimal selling thresholds would be higher and the buying thresholds would 
be lower than what we have derived in this paper and the "do nothing" range would be 
wider. Qualitatively, this would mean an overall choppier response from storage, with high 
elasticity over a narrow range of prices and low elasticity over a wide range, which would 
be undesirable from a system operation and reliability point of view. 

While our paper largely focused on the economic value of storage, it is important to 
recognize and quantify the environmental and the reliability value of storage. With proper 
control policies, storage can help matching stochastic supply with demand, improving sys- 
tem frequency and voltage profiles, and possibly mitigating large blackouts. The develop- 
ment of a systematic framework for quantifying the value of storage, the trade-offs between 
reliability, environmental, and economic value of storage, and the design of real-time pric- 
ing and market mechanisms for optimally striking these trade-offs are fundamental and 
important directions for future research. 
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7. Appendix 
Proof of Theorem Q} 

Proof. This proof proceeds by induction. Let us for the moment assume that the value 



function V k (-) = E [J k (•)] has the form defined in (12). 

From the dynamic programming algorithm, for k < N, we have 

Jk (s k ) = h k (s k ) + min _ _ {X k v k + E[J k+1 (s k + v k )}} 

u fe e[max(-s fc ,— v),v] 

where the penalty functions h k (s k ) are as defined in ([9]). 



(A.l) 



Then, we use the general form in (12) for E [ J k (s k )] and the state evolution equation in 



(|2j), and let i G Z + be such that v k + s k E [iv, (i + l)v). Applying the induction step to ( A.l ) 
we obtain: 



Jk{s k ) = h k (s k ) + min __ {\ k v k - tf k+1 (s k + v k ) + 4+i} 

v k e[max(-s k , —v),v] 



(A.2) 



Solving the optimization problem in (A.2) yields the optimal policy shown in (10). Now we 



can plug in the optimal policy to obtain J k (s k ). Let i G Z + be such that s k G [iv, (i + l)v). 
If < s k < v, then 



Jk (Sk) = 

and if s k > v, then 

Jk {Sk) : 



(A fc 



X k )s k + el +1 + c 



ntl +1 <x h 



+ (h° k -X k )s k + el +1 + c° k tftl +l <\ k <t 
(A fe - t£+ 1)*> + - tl +1 )s k + e\ +l + c° k if X k < t\ +l 



k+l 



[X k - t{ +1 )iv + {h{ - X k )s k + e| +1 + c\ „ fc+1 ^ „ k ^ , k+1 

(A* - t k +\)(i + l)v + (h{ - X k )s k + e k + + \ + 4 if tj+J < A fe < 4 +1 



if it^i < A fc < 



Let us recall that for a piecewise function / defined according to 

[fx (r) x + g± (r) if r < a 
[f 2 (r) x + g 2 (r) if r > a 



<f i+i 
— L k+i 



we have 



E [f (x) 



Also, note that: 



: E [/ (x) \r < a] P (r < a) + E [/ (cc) |r > a] P (r > a) 
+ E [fifi (r) |r < a] P (r < a) + E [t/ 2 (r) \r > a] P (r > a) 

a 

E[/(r)|r<a]P(r<a) = £)/(r)P*(0) 

u • min 



(A.3) 



(A.4) 
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Now, let us apply (A.3) and (A.4) to the equations derived above for J k (s k ), and compute 
E [Jk (sk)] for k < N, which leads to the following results: 

if < s k < v, then 

E [Jk (a*)] = eg - s k [t\ +1 F k {t\ +1 ) -h° k + £ 6>ft(6>)] 

4 + i<«<^ mx 

if s& > then 

E [J* (a*)] = 4 - flJfc [i£i (1 " ^ (4+\)) + 4S** (4S) " ?4 + E BP M 

j.i+1 < 0<-fi-^ 

where e\ denotes the sum of the terms that have not been multiplied by s k . Hence, the 
thresholds for k < N and i G Z + are given by: 

t° k =ti +1 F k (ti +1 )-hi+ Yl ep >w 

tl +1 <e<\r x 

4 = 4+\ (i - F k (4+\)) + i + + \F k (tj+y - /»* + E 6>n(6>) 

c+l <6l ^*fc+l 



After some algebra, the above results can be written in the form presented in (11). The 
derivation of the intercepts e\ follows the same procedure. 

The next step is to verify, using induction, that the thresholds at each stage (i.e. t\) are 
non-increasing functions of i. Considering that t k-1 has a different general form than t l k _ 1 
for i > 0, we first need to show that t\_ x < t\_ x assuming that t k +1 < t\ for all i. Hence, we 
need to first show that 

4^-i(4)-^-i+ E ^-iW>^(i-^-i(4))+4^-i(4)-^Li+ E opk-i(0) 

Knowing that h\_ l > h Q k _ ll we can remove — h],_ l and —h k _ l from both sides. Then, by 
writing the cumulative distribution functions as summations, and rearranging some terms, 
the above can be rewritten as: 

4E Pk-i(o)+J2 p fe-iW4+E m-i(e)>t° k + J2 ^-iW*fc+E 

\™\<e<t° Ajfi»<e<tj ti<e<\%™ Aj>i»<e<tg *2<»<*2 

By breaking each summation into disjoint intervals, and factoring all the terms in the 
same interval and merging them into one summation, the above can be rewritten as: 

4 < E ^-iW(4+4-4)+E ft-i Pk-i(o)ti+ £ eiv-iW 

a™<6»<^ t 2 k <e<ti t\<e<tl tl<e<\£^ 

We can see that in the equation on the right hand side (RHS) of the inequality shown 
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above, all the terms that have been multiplied by Pk-i (0) inside the summations are 

^raax 

greater than or equal to t k ; we also know that E fc_1 m . n Pfc_i (9) = 1. Hence, we verify by 
inspection that the RHS equation in the inequality shown above is always greater than or 
equal to t k . 

Finally, we have assigned a non-positive cost of —t to each unit of energy left in storage at 
stage N (i.e. E [J N (s N )] = —is N for s N < s). Also, a very small (negative) value is assigned 
to the thresholds for i > n at stage AT, to make sure we will not exceed the storage capacity 
in this stage. Hence, t l N is a non-increasing function of i, and t 1 ^ 1 < t l N is satisfied. 

We must now verify that t k + \ < t\_ x for i G Z + assuming that t\ is a non-increasing 
function of i. So, we need to verify: 

t\ (i - Fk-i (4)) +4 +2 ^-i (4 +2 ) - K + -\ + E (0) < ti 1 (i - F fc _ x 



,1 + 2 



<0<tl 



*r < e <*r 



Knowing that > h\_ x , we can remove — h l k \ and — /i* fc _ 1 from both sides. Then, by 
writing the cumulative distribution functions as summations, and rearranging some terms, 
the above can be rewritten as: 

E Pk-i(o)+ti +2 e n-i(<9+ E ^-i(0)<4 -1 +4 E ^-iW 



a™<9<4+ 2 t[ +2 <e<ti 



e E 



A™<e<4 +1 t l fc +1 - 



By taking all the summations to the right hand side and taking t\ to the left hand side of 
the inequality, breaking each summation into disjoint intervals, and factoring all the terms 
in the same interval and merging them into one summation, the above can be rewritten 
as: 

K x -A> E wm4 _1 -*i-(4 +1 -4 +2 )) + E ^-i(m4 _1 -4-(4 +1 -0)) 

\l%<e<e+ 2 ti +2 <e<t i + 1 

+ E Pk-i{e){t k - 1 -ti)+ E Pk-iWi^-e) 

t\ +i <8<ti 4<0<*l _1 

We can see that in the equation on the RHS of the inequality shown above, all the terms 
that have been multiplied by Pk-i (0) inside the summations are less than or equal to 
t\~ x — t l k ; we also know that E^ min^ -1 (^) — gi ven that the summations in the 

RHS do not overlap, we can verify by inspection that the RHS equation in the inequality 
shown above is always less than or equal to t l k l — 1\. 
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The last step is to show that the value function is a continuous function. This can be done 
by induction. We have defined E [J/v (sjv)] in such a way that it is convex. We also have 
defined h k (s k ) to be convex for all k. Looking at the equations of Jk(sk) in ( A.lj ) and (A. 2), 
given that E [Jk+i (sfc+i)] is convex, one can observe that the continuity of Jk(sk) is satisfied 
because v* k is a continuous function of s k . Similarly, considering the equations obtained for 
v* k , the continuity of E [J k (s&)] for all k follows from the fact that the expectation (i.e. 
E [J* (sfc)]) is simply a convex combination of continuous functions. This completes the 
proof. □ 

Proof of Theorem 

Proof. The results of Theorem [T] establish that for any finite N, the optimal policy is a 
threshold policy regardless of the price distribution. It can be verified that the form of the 
optimal policy extends to the infinite-horizon case in the sense that the infinite-horizon 
optimal policy is similar to the policy defined in Theorem [TJ with stationary thresholds 
that are only a function of the storage state. Furthermore, the optimal average cost is 
independent of the initial state. Hence, without loss of generality we may assume that we 
start from so = 0, and therefore, the storage state would only take values that are integer 
multiples of the ramp rate, i.e., s G {iv | i 6 0, 1, • • • ,n}. This assumption simplifies the 
optimal policy to either buy as much as v, or do nothing, or sell as much as v. 
For conciseness and ease of notation, and without loss of generality, we present the remain- 
der of the proof for the case of v = 1, A m i n = 0, and A max = 1. The proof for the general 
case is similar. Let t(s) and t(s) denote the threshold associated with selling and buying, 
respectively, at state s. Thus, if the stage price X k falls within the interval (t(s), t(s)), the 
decision is to do nothing and no cost is incurred. Let J k denote the cost at stage k. Then: 

E[J fc ] = E [E[J k \s k ]} = E[E[A fc min{s - s k , l}\s k , X k < t(s k )}P(X k < t(s k )) 
+ E[A fc max{-s fe ,-l}|s fe ,Afc >t(s k )]P(X k >t(s k ))] 
> E[E[A fc max{-s fc , -l}|a fc , A fc > t(s k )]P(X k > i(s k ))}} 
>-P{s k >l)E[P(X k >t(s k ))] 

= -(l-P(a fc = 0))P(A fc = l) (A.5) 

where the first inequality follows from A m ; n = 0, and the second inequality follows from 
Amax = 1 and the fact that selling can happen only when i ^ 0. Thus, E[J k ] is bounded 
from below by (A.5), with exact equality holding for a two-point distribution with PMF: 

a if A = A max = 1 

F\(\) = {l-a if A = A min = 
otherwise 
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a a u 

Figure 10 Underlying Markov chain associated with storage state under two-point price distribution 

Next, we show that a = 0.5 minimizes the stage cost. Under the two-point distribution 
defined above, the storage state evolves according to the discrete time finite-state Markov 



chain shown in Figure 10 



Letting pi denote the steady state probability of state i, it can be verified that p = 

1/(1 + 6 H \-b n ), where b= (1 — o)/o. Considering that the optimal decision is to either 

purchase at a price of A min = (which incurs a cost of zero) or sell at a price of A max = 1 
(which occurs with probability a(l — p ) and incurs a cost of — A max tJ= —1), we have 



7 



lim 

JV->oo 



1 

N 



E 



'JV-l 



.fc=0 



JV-1 



X k v k | s = 

— J^E[E [A fc max{-s fc ,-l} | X k > t(s k ), s k ] P(X k > t(s k ))] 

k=0 

. JV-l . JV-l 

= - lim ^E^^w^^-^^Ei 1 



lim 

JV^-oo 



k=0 



Po)a, 



fc=0 



which yields 



7 : 



6(1 + 6H H6' 



,n-l> 



(6+l)(l + 6 + -.- + 6 n ) 



(A.6) 



Solving for b to minimize 7 yields 6=1, which corresponds to a = 1/2. This proves (17) 



(18), and (21). It remains to show that the differential cost function corresponding to (17) 



is of the form (19). 



Let H(s) be the value of the right hand side (RHS) of equation ( 16 ) obtained by plugging 

! have 

i(i + l)(A max — A m j n )' 



the proposed solution for H*(s) and 7* into equation (16); then we have 

(i + l)X min + (n-i)X 



H s =E 



+ 



min Xv - 

v £ [max( — s , — v ) , min (v,s— s ) ] 



71+1 



'-(s + v) 



2(n + l) 



v(Xr 



Amir 



n 



71 + 1 



(A.7) 

where iv < s + v < (i + l)v, for alH £ {0, 1,2, • • • , n — 1}. 

Therefore, we need to show that H(s), obtained from solving the optimization problem 



in (A.7), is indeed the H*(s) defined in (19). It can be verified that the solution to the 



optimization problem in (A.7) is as follows: 

max(-s, -v) if A = A max 
miniv, s — s) if A = A min 



(A.8) 
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It can be verified by inspection that if we plug (A. 8) into (A. 7), the H(s) obtained is indeed 
the H*(s) defined in ( Jl9| ). The proof is complete. □ 

Proof of Corollary [1} 

Proof. Since a two-point distribution with non-zero probability masses placed at the 
endpoints of the fixed support is always achievable for any \x e (A m ; n , A max ), the additional 
assumption of a fixed mean does not affect the proof of Theorem [2] up to and including 
equation ( A. 6 ). For the proof of this corollary, we need to minimize 7 in equation (A. 6 ) with 
respect to b subject to the constraint \i = 6A m ; n / {b + 1) + A max / (& + !)• This optimization 
problem has the unique solution b = (A max — jj)/(fj, — A min ) . Recall that a — l/(b + l). Hence, 



a - 



(fx — A min )/(A max — A min ). This completes the proof. □ 



